Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rendering and recording mirrored video stream from camera. #1628

Open
wants to merge 11 commits into
base: gh-pages
Choose a base branch
from

Conversation

stepancar
Copy link

@stepancar stepancar commented Oct 4, 2023

Description
Added a demo which demonstrates rendering and recording mirrored video stream from camera.

Purpose

Mirroring video is a common case in video processing.
#1626
This demo shows advantage of processing video frame-by-frame in a worker
https://stepancar.github.io/webrtc-samples/src/content/insertable-streams/video-recording/

@hthetiot
Copy link

hthetiot commented Oct 24, 2023

@stepancar I was wondering if you could add the work around to make MediaRecorder recorded video seekable part the of sample. Doing so would inform developers that implement recording of the issue and potentially serve as reference to push Chromium team reconsider the Won't Fix state of the bug (I can always dream) if all other non chromium browsers proves not having the issue.

I would understand if you refuse, at least mentioning that recording file may not be seekable from this sample when using Chromium based browser can in my point of view go in the right direction and help the cause.

Chrome related WontFix bug:

Know working workaround:

@stepancar
Copy link
Author

@hthetiot , I would be happy do that too, but when I get any comments from maintainers to make sure this sample makes sense

@stepancar
Copy link
Author

@fippo , could you review it? Thank you

@stepancar stepancar changed the title Rndering and recording mirrored video stream from camera. Rendering and recording mirrored video stream from camera. Jun 21, 2024
@stepancar
Copy link
Author

Hello @fippo! May I ask you to review it/validate idea?

let recorders = [];

startButton.addEventListener('click', async () => {
const stream = await navigator.mediaDevices.getUserMedia({ video: { width: 1280, height: 720 } });
Copy link

@hthetiot hthetiot Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard coded camera size will fail on many device.

Edit: you do part bellow already:

Use video: true constraints and videoWidth/Height on video tag once metadataloaded to get size instead or VideoTrack width or height settings.

@stepancar
Copy link
Author

@hthetiot , do you know how to get a feedback from maintainer in this project?

@fippo
Copy link
Collaborator

fippo commented Jul 27, 2024

@stepancar apologies for the radio silence. The samples repository is effectively in maintenance mode and hasn't had a dedicated maintainer for a while.

The use-case for the sample is interesting, one normally does not think much about this as most mirroring is done with CSS but that does not apply to MediaRecorder. But mixing mirroring with MediaRecorder and then providing two modes makes this way more complicated than it would need to be. It would probably fit better as a processor in the video-processing sample but that one is not using a worker and the code makes me cry.

If you still see the slowdown filing a specific crbug for it would be better.

@stepancar
Copy link
Author

@fippo , thank you for your answer!

I know that usually people mirror video with css. This works well when we need to implement video calls app.
But here I'm trying to demonstrate a specific scenario - recording mirrored video. Which is a use case for video editors.

This demo helps to better understand limitations of MediaRecorder API / canvas.captureStream and demonstrates how https://github.com/w3c/mediacapture-transform can help here.

Yes, know about https://webrtc.github.io/samples/src/content/insertable-streams/video-processing/
I thought to add it there, but there is already and existing canvas 2d example. I would say it's a different kind of demo
https://webrtc.github.io/samples/src/content/insertable-streams/video-processing/ - is saying if you want to transform your stream you can do different things like... . The problem with that - it does not demonstrate why should I do it. all transformations from that demo could be implemented without https://github.com/w3c/mediacapture-transform
My demo demonstrates problem (frame drops in recorded video) and shows the solution.

I didn't get your point about crbug, could you elaborate on that? Media recorder behaviour follows specification.
Thank you!

@fippo
Copy link
Collaborator

fippo commented Aug 5, 2024

I get the use-case and it is great! But adding it to the simple recording demo is out of question because iirc Firefox does not support breakoutbox yet (and don't even try to get started with worker vs non-worker) and it brings in a ton of complexity.

Now if you actually look at the problem that needs solving it would be way more efficient just to use libyuv's I420Mirror method in-place rather than doing multiple thread hops etc. Works nicely but requires a JS API change to add a flip option.

Over to @guidou

@guidou
Copy link
Contributor

guidou commented Aug 6, 2024

@stepancar Can you file a spec bug at https://github.com/w3c/mediacapture-main/issues/ or https://github.com/w3c/mediacapture-record/issues explaining the use case and, if you have a proposal, a sketch of it and its rationale?

@stepancar
Copy link
Author

stepancar commented Aug 6, 2024

@fippo , I totally agree with you that it would be nice to have simple flip option in media capture api, or, even in api where we get the stream
https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia
It would be more efficient and convenient for this specific use case

but the problem is that those api are not about transforming streams and introducing such option there is abstraction/responsibility leaking.
So, not sure maintainers will want to support it (I will ask in another issue)
The same time https://github.com/w3c/mediacapture-transform is designed for that, right?
This is why I'm adding this demo here

Ideally, it should be another, non javascript api for making such transformations, without requiring to write js code and thinking about workers, etc.
Something like

const mediaStream = new MediaStreamTransform(originalMediaStram, {
     transform: 'scaleY(-1)', // https://developer.mozilla.org/en-US/docs/Web/CSS/transform
     filter: 'blur(5px)'  // https://developer.mozilla.org/en-US/docs/Web/CSS/filter
})

Talking about firefox is not supported. I'm not sure how this can affect this sample. this sample is for https://github.com/w3c/mediacapture-transform section. If it's not supported by vendor - it's expected that it would not work.
It means, existing demos which use breakoutbox also don't work in FF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants